Add 0082 Langfuse failed-Generation rendering#232
Merged
Conversation
PR C of proposal 0082, completing the 3-PR split with the Langfuse failed-Generation rendering. 0082 is now fully implemented. For a structured_output_invalid failure the bundled Langfuse observer renders the response-side surface on the ERROR Generation like a completion: output (payload-gated) + usage on the Generation, and finish_reason / response_id / response_model in metadata (the completion-only isinstance guard widened to include the category), alongside the existing level=ERROR + statusMessage=category. Every other failure category is unchanged. Fixture 123 is driven in the dedicated Langfuse harness (response_schema threaded + expected_error handled). conformance.toml marks 0082 implemented. Also folds in the whole-proposal adversarial review over all three PRs: - Cross-observer consistency: the failure event builders project empty output_content to None (matching the success path's content-or-None), so both observers omit empty content identically. The OpenAI wire cannot currently produce empty content on the structured path, so this is a defensive builder projection with a builder-direct test. - The Langfuse conformance harness now asserts observation statusMessage (previously a silent no-op, so 123's category assertion did not run; also affected fixtures 130/137/138). - Refreshed a stale _typed_event_metadata docstring, dropped a docstring spec ref, and completed the CHANGELOG response id/model listing.
There was a problem hiding this comment.
Pull request overview
Completes proposal 0082’s observability work by rendering structured_output_invalid response-surface details on Langfuse ERROR Generations (mirroring completion rendering and OTel parity), and updates conformance + release notes to mark 0082 fully implemented.
Changes:
- Render payload-gated
outputplususageon Langfuse failed Generations forstructured_output_invalid, and emit response-side metadata (finish_reason,response_id,response_model). - Extend Langfuse conformance harness to drive fixture 123 via
response_schemaand assertstatusMessage; add focused unit tests for rendered vs redacted output. - Ensure empty structured-failure content is projected to
Nonein OpenAI event builders for cross-observer parity; mark 0082implementedinconformance.tomland updateCHANGELOG.md.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/test_observability_langfuse.py | Adds unit tests covering Langfuse failed-Generation rendering (payload on/off). |
| tests/unit/test_llm_provider.py | Pins builder behavior: empty structured-failure content projects to None. |
| tests/conformance/test_observability.py | Keeps case 123 out of the generic observability harness with updated rationale. |
| tests/conformance/test_observability_langfuse.py | Drives Langfuse fixture 123 via structured path; strengthens observation assertions (incl. statusMessage). |
| src/openarmature/observability/langfuse/observer.py | Implements Langfuse ERROR Generation rendering for 0082 response-surface on structured_output_invalid. |
| src/openarmature/llm/providers/openai.py | Projects empty raw_content to None in failed + retry-attempt typed events. |
| conformance.toml | Marks proposal 0082 as implemented and updates the status note. |
| CHANGELOG.md | Updates 0.17.0 entry to include Langfuse rendering details for 0082. |
Address CoPilot review threads on PR #232 (0082 Langfuse failed-Generation rendering): - Assert metadata.response_id on the failed-Generation unit test. The event set it but only response_model was pinned, so a regression dropping response_id would have passed. - Assert raised_from against the raised NodeException in the linear conformance path. Fixture 123 declares it but nothing checked it. Category is not asserted on the exception (its category is always node_exception); it rides on the Generation statusMessage. - Restructure renders_response_side to an explicit if/else so the failure branch is obvious. No behavior change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR C of the 0082 split (structured-output failure diagnostics, spec v0.77.0), completing the proposal. PR A (#229) landed the event + error surface; PR B (#230) the OTel error-span rendering + the token metric; this closes it with the Langfuse side.
conformance.tomlnow marks 0082implemented.What
For a
structured_output_invalidfailure, the bundled Langfuse observer renders the response-side surface on the ERROR Generation like a completion:output(payload-gated) +usageon the Generation;finish_reason/response_id/response_modelin metadata (the completion-onlyisinstance(LlmCompletionEvent)guard in_typed_event_metadatawidened to include the category);level = "ERROR"+statusMessage= the category.Every other failure category is unchanged. Fixture 123 is driven in the dedicated Langfuse harness (
_LANGFUSE_FIXTURES), which I taught to threadcalls_llm.response_schemaand to catch a raisingexpected_error. Two focused unit tests (rendered / redacted under payload-off).Whole-proposal adversarial review (folded in)
After all three PRs were done, I ran one adversarial pass over the entire 0082 implementation (cross-observer parity, the retry path,
error_messageconsistency). Findings folded in:output_contenttoNone(matching the success path'scontent or None), so both observers omit empty content identically rather than one rendering""and the other dropping it on a truthiness gate. The OpenAI wire can't currently produce empty content on the structured path (an empty assistant message with no tool calls fails earlier asprovider_invalid_response), so this is a defensive builder projection, verified by a builder-direct test._assert_observationnever checkedstatusMessage— so 123's category assertion (and 130/137/138's) was a silent no-op. Added the assertion._typed_event_metadatadocstring, dropped a docstring spec ref, and completed the CHANGELOGresponse_id/response_modellisting.Full suite green (1716 passed), pyright clean.